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(54) Device and method of im^ 



(57) The irivention relates to a process for temporal 
interpolation with motion compensation of an image j lo- 
catedl between an image n-1 and an image n, making 
an estimate bf motion (1) between the image n-1 and 
the image n to provide a field of motion vectors V(n-1, 
n). and interpolating (2) the image j as a function of this 



motbn vector field, characterized in that it detects (3) 
defective motion vectors and In that it performs low-pass 
filtering (4) over zones of the Interpolated image corre- 
sponding to these defective vectors. 

The applications are in the field of image process- 
ing, for example conversion between standards. 
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[0001] The invention relates to a process and to a device fcr tennporal image interpolation with motion compensation, 
employed in the processing of television image sequences. It is more particularly designed for motion estimators of 
the recursive PEL type, that is to say ones based on calculating the motion at the pixel level rather than at the image 
block level 

[00021 The possibility of calculating motion at the pixel level has made it possible to improve the quality of interpolation 
systerro considerably. However, the motbn cannot be measured at all points in the image: when a plurality of objects 
are moving differently in a scene, for example motion of a first plane relative to a background plane, pixels appear or 
disappear from^ one image to another which makes it impossible to measure their movement. The presence of these 
zones ir^.ierferes with the motion estimator, and the resulting defects are particularly problematic. These zones require 
particular treatment to suppress or at least mask the defects. 

[00031 An interpolation process having motion compensation with a fallback solution is known, having been described 
in French Patent No. 8904256, The process detects. the defective nru^tion vectors by means of a local operator of the 
variance type. The mbtion vector field is then modified by forcing the defective vectors to the value zero The interpo- 
lation wrm motion compensation is then applied with this new motion vector field: 

[00041 A pixel labelling process is also known f rom French Patent No. 91041 64, classifying them as normal, appear- 
ing disappearing or occluding pixels, for particular processing as a function of how they are labelled. The interpolation 
is based on the temporal projection of the rnotkxi vector field and is adapted to this pixel classification. 
[0005] Although the first process is simple to use. the obtained image quality is not always salisfacloiy. The second 
process is much more complex, and requires estimation of the motion fields of the input images. The improvement 
obtained with a process of this type is not commensurate with the complexity which ft entails. 
[0006] The object of the present invention is to overcome the aforementioned drawbacks. 

[0007] The invention relates to a process for temporal interpolation with motkxi compensation of an image j located 
between an image n-1 and an image n. making an estimate of motion between the image n-1 and the image n to 
provide a field of motion vectors V(n-1 . n). and interpolating the image j as a function of this motion vector field, char- 
acterized in that it detects defective. motion vectors and in that it performs low-pass filtering over zones of the interpo- 
lated irriagecorrespwding to these defective v^^^ 

[0008] ^ It also relates to a device for impiementirvg the process described above, comprising amotion estimator which 
30 receives an image, n-1 and an image n in order to calculate a field of motion vectors V(n.1 , ri). and an interpolation 
circuit Wiich calculates an interpolated image Y(j) from these two images h-1 and n and the motion vector field, char- 
acterized in that it also includes a circuit for detecting defective motion vectors, which receives the motion vector field, 
and a IcW^ss filtering circuit for filtering the interpolated image as a function of the information transmitted by the 

detectkyi circuit. . . ■„ 

35 [0009] The invention will be clearly understood, and its advantages and other characteristics will become more ap- 
parent during the following descHptbn. given by way of example and with reference to the appended figures, in which: 

- Figure 1 represents a diagram of the devrce according to the invention; 

. Figure 2 represents the morphologk^al operations performed on the image with the points having defective vectors 
40 , marked. 

[0010] By virtue of the invention, the zones where the interpolation with motion compensation is defective are proc- 
essed satisfactorily. The process is simple to, employ It gives better results than are known from the prior art with 
equivalent. complexity. For comparable processing, its image quality Is improved. 

45 [00111 Figure 1 represents a device for implementing the process according to the invenlkjn. 

[0012] The two inputs of the devrce are connected in parallel to a motion estimator 1 and to an interpolation circuit 
2 The output of the motion estimator 1 is connected in parallel to the input ot a defect detection circuit 3, to a third 
input of the interpolation circuit 2 and to a first input of a tow-pass filter 4. The output of the interpolation circuit 2 is 
connected to a second input of the low-pass filter 4. The defect detection circuit 3 is connected to a third input of the 

so low-pass filter 4. The output of this filter is the output of the device. 

[001 3] The digitally encoded luminance information Yn-1 , for the preceding image n-1 . and Yn for the current image 
n are present simultaneously on the first input and the second input of the device. An estimate of motton is nnade 
between these two images. This motion estimate may be of the recursive PEL type. A nnotton vector is then assigned 
at the level of each pixel, thus creating a fiekJ of motion vectors V(n-1, n) for the image j to be interpolated which is 

55 located between the images n-1 and n. This vector f iekJ is transmitted, amongst other things, to the third input of the 
interpolation circuit 2. On its first two inputs, this circuit receives the infomiation Yn-1 and Yn. On the basis of the motion 
information coming from the field of motkxi vectors V{n-l. n). and without taking account of a possible deficiency of 
the motion vectors, which defteiency wai be defined later, an interpolated luminance image YQ) is cateulated and trans- 
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mitted to the lowiDass filter 4. . ^. . . ^ . . .i. 

[0014] The vector field is also transmitted :o a defect detection circuit 3 The function of this circuit is to detect the 
defective motion vectors. This detection uses measurement of the spatial incoherence of the horizontal and vertical 
components of the vector field. • 
10015] This measurement may be perlomr-ed by a variance calculation on each component of each vector of the 

current vector field (temporal index t). c ^ 

fOOiei In one illustrative embodiment, the calculation, for a given pixel, is carried out on a window of 5 x 5 pixels 
centred around this pixel. The mathematical equation of this variance calculation is then given by the following equa- 
tions: 



^0 ^ f ^ 



2S 



30 var.vv(<.y,/)=^2; 2 



km-t tm-i 



[001 7] v(i, i, t) represents the motion vector assigned to the pixel of column i and row j for the image t. 
35 [001 8] vx(i j t) and vy(i, j, t) are the horizontal and vertical components of the motion vector v(i, j, t). 

100191 moy v(i i t) is the average value of v over the window centred around the pixel i. j for the image t. 
0020] The resjits var_vx(i. j. t) and var_vy(i, j, t) are then compared with a threshold. If one of the 2 vanances 
exceeds the threshold, the vector is then marked as defective. This threshold is. for example, taken as equal to 1 0. A 
binary image of the processed vector field is then obtained, in Which the binary level '1 ■ denotes the detective vectors 
40 and the level 'O' denotes the non-defective vectors. ^ . , • . .u • 

r00211 This binary image is then processed by a mathematical morphology filter in order to eliminate the isolated 
markings and expand the zones of defective vectors. This processing, described betow. is also carried out by the error 
detection circuit 3 Figures 2a. 2b. 2c represent the actions of the various operations on the pixels of an image. 
[00221 A first processing operation is filtering of the erosron type on the basis of a sliding structuring element covering 
46 the binary image previously obtained. This structuring element is a window or image block of determined size, in our 
example a block of size 3x3 pixels. If, tor a pixel in the image marked with -r. at least four neighbouring pixete 
belonging to the structuring element are marked with -I', then this marking ('V) is retained, otherwise it is set to 0 
[00231 Figure 2a represents an example ol a binary image portion obtained after processing of the vector field The 
asterisks present in the boxes symbolizing the pixels correspond to the vectors marked as defective by thresholding 
so the variance Figure 2b represents the binary image obtained after erosion by a structuring element of size 3 X 3 pixels. 
The asterisks whch have been crossed out correspond to the pixels whose marking is suppressed after the morpho- 
logical erosion operation. They correspond to the value -O'. and the renr«ining asterisks correspond to the value 1 
[<»241 The second processing operation, carried out on this new image, is filtering of the expansion type. The struc- 
turing element used is. for exanple. a block of size 3 x 3 pixels. It. for a pixel marked -O". at least one neighbouring 
55 pixel belonging to the structuring element is marked with '1 •. then the marking of this pixel is forced to •2-. 

[002S1 Figure 2c represents the ternary image obtained after expansion carried out on the eroded image. The struc- 
turing element has the size 3x3 pixete. The circled asterisks appearing at the border of the asterisks which have not 
been crossed out from the preceding image correspond to the pixels obtained after expansion, and therefore to the 
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[^^^ Thrte'mary mask obtained in mis way, available at the output of the defect detection circuit, is transmitted to 

iMaTl'^lfindicaL above, the interpoteted image is transmitted onto a second input of the low-pass filter 4, which 
also receives the motion vector field or, a first input, and the ternary mask on a third input. 
[0028];. the low-pass filtering is carried out on the defective zones in the following way: 

. , if th-ecurrent pixel is marked 'Or nc filtering is appliedto the corresponding pixel of the interpolated image received 

- rtfleolrrent pixel is marked -1 ". low-pass filtering is applied to this pixel over a zone of size 5 X 5 pixels around 

the corresponding defective pixel in the received image 

-. ^,hecurr£tpixerismarked•2^low-passfineringisappliedtoazoneofsi^ 

defective pixel in the received image. 

r00291 The sizes of the filtering windows are given by way of example. The less severe Iowi)ass filtering applied at 
l^^riphery of- the defective zones (ppceis marked "2-) permits soft transitbn between the defective pocel zones and 

IS^S^^^^ the:value of the variance which is associated with the vector and measures 

S. S'iS^.^ is then emp^yed to determine the coefficients of the iow-pass finer, thus 

20 modifvind its response as a function of the confidence attributed to the cur^^ ' ^ • , 

the Chrominance values; as well as the luminance values. The irUerpolation 
circuit^then calculates th^ interp)blated chrominance images, on which these filtering operations ^^^^^^"^^^^^^ 
Si The fl^lds^f application of the invention relate, for example, to frequency conversion, changing between 
standards or any application involving temporal image interpolation on the basis of motion compensation. 
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Claims 



Process for temporal Interpolation with motibn compensation of an image j located between an '"«9e n-1 and an 
^ge h: ,^^3king^n estimate of rrotion (1 ) between tKe image n-1 and the image n to provKle a field of rnot.or^ 
vedk,rsV(rtVl,n); and Interpolating (2) the image j as a function of this motion vector field, c^aracterced «i thatrt 
detects ^y^defectlve motion vect^ that it performs low-pass filtering (4) over zones of the interpolated 

image corresponding to these defective vectors. 

Process according to Claim 1 . characterized in that the defective motion vectors are determined as a function of 
varlahce^information measuring the kxal spatial incoherence of the motion vectors. 

Process aiSrtling ^ Claim ^. characterized in that the filtering is adapted to the value of the variance. 

Process according to one of the preceding claims, characterized in that the detection of defective motkxi vectors 
is tollow^d by rriprphological eroskxi arid expansion opera 

6. jF-rdCessabbordingtot 

are winidbwS of siz6 3 X 3 pixeb^ 

6 Process according to Claim 4 or 5 characterized in that the filtering window empkDyed for t^^^ 'ow-pass filte*g is 
of a ^ller size tor the pixels marked after the expansion operation than is used for the pixels whose marking is 
obtained after the erosion operalkjri. 

7 Process according to Claim 6; characterized in that the size of the filtering windowfor the pixels v^ich are obtained 
after erosion is 5 ^5 pixels, and that of the one used for the pixels which are obtained after expans»n « 3 x 3 pocels. 

8. Process according to ar^ one of the preceding claims, characterized in that the interpolation and fiMering are 
apjjlied t6 the chrominance images. 

9 Device for implementing the process according to Claim 1 , comprising a motion estimator (1) whteh receives ari 
S n-t anS an image n in order to calculate a fieW of motion vectors V(n-1 . n). and an *nterpolatjc« circuit (^^ 
r^hcalculates an inrerpolaied image YO) from these two images n-1 and n and the moton vector fieW. charac- 
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terized in that it also includes a circuit (3) for detecting detective motion vectors: which receives the motion vector 
field, and a low-pass filtering circuit (4) for filtering the interpolated image as a function of the information transmitted 
by the detection circui: (3). 

5 10. Device accoiding to Claim 9. characterized jn that the detection circuit (3) detects the defective motion vectors on 
the basis of variance information. 

11 Device according to Claim 9 or 10. characterized in that the circuit (3) for detecting defective vectors supplies a 
' binary image identifying the pixels relating to the defective motion vectors, and in that it performs morphological 

10 erosion and expansion operations on this image in order to supply a ternary image to the low-pass filter (4). 

12 Device according to Claim 9, 10 or 11. characterized in that the low-pass filtering of the interpolated image is 
" carried out on a filtering window with smaller dimensions for the pixels which are marked after the expansion 

operation than for the pixels whose marking is obtained after the erosion operation. 

13 Device according to Claim 10. characterized in that the filtering circuit (4) determines the coefficients of the low- 
pass fitter as a function of the variance measuring the local spatial incoherence of the motion vectors. 

14. Device according to Claim 9, characterized in that the interpolation circuit (2) interpolates the chrominance images, 
20 and in that the filtering circuit filters these interpolated chrominance images. 
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